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Abstract 


Microservices Architecture is a recent approach to decomposition of distributed systems. Many 
disparate microservice-based systems exist in industry. However, there are currently no software 
suites available which facilitate rapid prototyping of these systems for the purposes of scientific 
evaluation. 


This project proposes, develops, and evaluates a system for rapid prototyping of microservice based 
systems. The emulation system is delivered as a highly configurable and extensible set of tools 
which can create production-grade clusters hosted on a leading cloud provider. 


The results of this work show that a system produced using the emulation tool set yields an accurate 
representation of a true microservice-based system composed of multiple interconnected services, 
with differing resource demands. 
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1 Glossary of Terms 


Cloud Provider — A vendor that provides Infrastructure, Platform or Software as a Service models 
of consumption. 


Cluster — A set of network-connected physical machines, virtual machines or containers, into 
which services are deployed and managed by a single cluster orchestrator. 


Cluster Orchestrator — A software system that automates upgrades, fault recovery, scaling and 
management for a set of services or applications running on containers or virtual machines. 


Container — An isolation technology which abstracts the operating system from applications 
running within the isolated unit. 


Infrastructure as a Service — A pricing model where infrastructure is provided and maintained by 
a vendor for a set price. 


Kubernetes — An open-source cluster orchestrator, originally developed by Google which manages 
container workloads [1]. 


Kubernetes Node — A virtual or physical machine capable of hosting microservices and controlled 
by master Kubernetes components [2]. 


Platform as a Service — A pricing model where an application platform abstraction layer is 
provided by a vendor and the vendor maintains the underlying software and hardware, for a set 


price. 


Service Fabric — An open-source cluster orchestrator, originally developed by Microsoft which 
manages container or virtual machine workloads [3]. 


Service Fabric Application — A Service Fabric idiom for encapsulation of related services or 
applications into a domain-specific logical deployment unit [4]. 


Service Fabric Node — A virtual or physical machine that is part of a Service Fabric cluster [4]. 
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2 Introduction 


Microservices Architecture is a variant of Service Oriented Architecture where communication 
protocols are lightweight, services are fine-grained and are independently deployable [5], [6, p. 95]. 
At the outset of this project there were no microservice topology emulation suites. This project sets 
out to deliver such a suite. A configurable cluster prototyping suite is devised and delivered as a 
set of libraries, service templates and infrastructure automation tools. The prototyping suite enables 
the provisioning of an application within a Service Fabric cluster, where the application is 
composed of one to many configurable microservices. The flexible configuration and infrastructure 
automation tools enable emulation of complex systems for prototyping and performance testing 
purposes in a production-grade environment. 


To ensure that the prototyping suite can accurately model a real system; a complete system 
composed of multiple interconnected microservices is created, and its performance profile is 
obtained. A system composed of interconnected services configured to emulate the real services is 
generated with the prototyping suite, and its performance profile is compared to the real system to 
verify its legitimacy. 


2.1 Report Layout 
This section provides a summary of the report chapters. 


Chapter 3 describes the design decisions and architecture of the Cluster Emulator prototyping 
suite. 


Chapter 4 describes the evaluation of the prototyping suite and the accuracy of systems generated 
with the suite. 


Chapter 5 represents the conclusion, along with potential areas of further development. 
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3 Cluster Emulator 


3.1 Overview 


Cluster Emulator is a toolset which enables rapid prototyping and reconfiguration of microservice- 
based systems in an on-premises or cloud environment. The primary goal of the toolset is to enable 
prototyping capabilities in this field for use in performance engineering and other research. 


At the core of Cluster Emulator is a library which facilitates registration and execution of real, 
emulated, and simulated business logic. This emulation library is utilised by other libraries and 
tools to enable generation and deployment of a fully operational cluster, based on a simple 
configuration file. 


The following sections detail: 


e design decisions related to technologies and other constraints, 
e the architecture of Cluster Emulator libraries and tools to meet these constraints, and 
e how this architecture enables the creation of prototype microservice-based systems. 


3.2 Design Decisions 


3.2.1 Openness and Extensibility 


The microservices space is continually evolving with each cluster orchestrator, cloud provider and 
operating system providing differing capabilities. To promote reuse and allow for evolution, the 
core of the project needed to be modular and cross-platform. At the outset of this project, there was 
no existing technology that enabled rapid prototyping of microservice systems with configurable 
performance and networking profiles. The project is built using well-maintained, open-source, 
components and core libraries are platform agnostic, allowing for maximum reusability. 


3.2.2 Language and Application Framework 


Due to the author’s technical competencies, C# is the implementation language for the core libraries 
and template service. To avoid any platform dependency, all projects and libraries target .NET 
Standard [7], .NET Core or ASP.NET Core [8] while all scripts target PowerShell Core [9]. YAML 
[10] is used to configure automated builds in the continuous integration pipeline. 


3.2.3 Cluster Orchestrator 


The prototyping suite would need to integrate with a cluster orchestrator to enable the evaluation 
of performance profiles of microservice-based distributed systems. [11] briefly describes several 
popular orchestration platforms at a high level. The two most prominent open platforms, Service 
Fabric and Kubernetes, were chosen for deeper comparison. Technical [2], [12], [13], [14] and 
comparative [15], [16], [17] literature show that both platforms meet all the project requirements. 
Service Fabric is the orchestrator chosen for this project as the learning curve is shallower, it 
provides more capabilities, and there was more local support available to the author. For hosting, a 
Platform as a Service (PaaS) solution was preferable to minimise time spent on configuring 
infrastructure. Azure [18] is the chosen hosting platform as it is the only cloud provider that 
provides PaaS support [19] for Service Fabric. A summarised view of the literature reviewed for 
cluster orchestrator comparison is presented in Table 1. 
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Availability 


Learning Curve 


Community 
Support 


Language 
Support 


Cloud Provider 
Support 


Ease of Use 


Capabilities 
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Service Fabric 


Open Source on GitHub [20]. (MIT 
License) 


Low for those with existing .NET 
experience. 


A large amount of expertise in the 
author’s place of work. 


Active community on prominent 
development forums [20], [22]. 


Broad language support with native 
integration for .NET. 


Supported via IaaS on all vendors. 


Provided as PaaS on Azure. 


A lot of platform-specific idioms, 
driven by technology. 


Well documented and SDK is 
straightforward. 


Integrated support in prominent 
IDEs and development tools. 


SDK provides much functionality. 


Provides application and container 
orchestration. 


Provides application models for the 
development of highly resilient 
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Kubernetes 


Open Source on GitHub [21]. 
(Apache License) 


Very high. 


Little expertise in the author’s place 
of work. 


Very active community on 
prominent development forums 
US esa 


Broad language support but 
requires container configuration 
regardless of language. 


Supported via IaaS on all vendors. 


Provided as PaaS on the top three 
cloud vendors. 


A lot of platform-specific idioms, 
driven by semantics or analogous 
container concepts. 


Well documented and SDK is 
straightforward. 


Support across many IDEs and 
development tools. 


SDK provides basic functionality, 
requiring integration and 
maintenance of external 
components for specific 
capabilities. 


Provides container orchestration 
only. 


All application and container 
configuration is left to the 
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Interoperability 


System 
Structure 


Configuration 


Fault Tolerance 


actors, stateful services or stateless 
services. 


Provides unified features for 
resilient communication, service 
monitoring. 


Runs on Linux or Windows. 
Opinionated platform and SDK. 


If an application takes deep 
dependencies on the SDK, they will 
become tightly coupled. 


Completely decentralised cluster 
state management and fault 
detection. 


Communication happens between 
cluster nodes positioned on a ring. 


System services are the same as any 
other stateful service and manage 
cluster state through quorum. 


Service Fabric Applications are 
configured using .xml files. 


High for stateless and stateful 
services. 


As the system is entirely 
decentralised, nodes can report state 
more regularly and be removed 
within a maximum of 30 seconds, 
though generally much sooner. 


Table 1 - Comparison of cluster orchestrators 


3.2.4 Telemetry 
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developer, creating a heterogonous 
ecosystem. 


Runs on Linux or Windows. 


Unopinionated platform geared for 
extensibility. 


Though possible to integrate with 
other container engines, it is 
effectively bound to Docker [24] if 
any support is required. 


Distributed, but centralised, cluster 
state and fault detection. 


Nodes individually report state to a 
separate API Server. 


The API server is a distinct entity 
that manages state using a separate 
etcd [25] key store. 


Container images are configured 
using .yaml files. 


High for stateless services. 


Low for stateful services, without 
careful consideration. 


Liveness probes test the health of 
services and trigger service 
removal. Unhealthy services are 
removed and are considered 
entirely dead after 5 minutes. 


To ensure that issues could be easily debugged, and performance metrics tracked during testing, 
application logs and system performance counters needed to be emitted to an external data store 
for processing and retrieval. 
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Serilog [26] was chosen as a logging abstraction layer for application-level logging to prevent 
logging infrastructure lock-in. Serilog is a widely used open-source logging library that supports 
structured event logging to many storage platforms and can be extended to support others. 


As Azure was the platform of choice, the most efficient way of handling application logging and 
analysis was to push the application logs to Application Insights [27]. This platform provides a 
complete data management solution, minimising the time spent on the setup of distributed tracing 
and log analysis. Application Insights provides all the required capabilities for service monitoring. 


The built-in Azure Monitor [28] tooling provided sufficient, coarse-grained system performance 
metrics to evaluate the prototyping system. However, to enable the fine-grained analysis of system 
performance and service health required to analyse the impact of nuanced configuration changes, 
operating system performance counters and Service Fabric logs would also be required. As both 
components log to disk by default, an agent was added to each node in the cluster to emit the 
necessary information to a data analysis solution. As this aspect of the project is specific to the 
chosen platforms, logs were emitted to Azure Log Analytics [29]. This platform takes care of the 
storage and querying requirements allowing efficient execution over multiple test configurations. 


3.2.5 Resiliency 


As web service request resiliency is central to the project, a resiliency solution that is well 
established, easily configured, open to extension, well supported and portable was required. Polly 
[30], an open-source .NET resilience and transient-fault handling library supported by the .NET 
Foundation, was found to meet these criteria. Polly provides many easily configurable resilience 
policies which are well documented. It is well integrated with the .NET HTTP stack and is also 
capable of being used independently of this allowing it to be easily adapted to suit non-standard 
use cases. 


3.2.6 Configuration 


JavaScript Object Notation (JSON) was chosen as the primary data notation for configuration files 
as it is an industry-standard and its key-value pair structure is simple for the end-user to understand. 
It can be easily validated both in a user’s editor and during parsing to objects. Newtonsoft.Json, an 
open-source library for interacting with JSON in .NET, was chosen to handle deserialisation of 
configuration files. It is widely used in industry and enables attribute-based mark-up for validation 
and inclusion of property values on deserialisation to an object instance. 
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3.3 System Architecture 


Cluster Emulator is composed of several modules. One core component of the project is a 
configurable, platform-agnostic, service emulation library. To evaluate a configured system in an 
Azure Service Fabric cluster, platform integration libraries and a service template consume the 
emulation library. A set of parameterised package generation, infrastructure provisioning and 
cluster deployment scripts automate the configuration and deployment process. As one of the core 
goals of the project is to ensure the prototyping capabilities are reusable, ease-of-use for the end- 
user was a primary requirement in the design of each component. 


Figure 1 shows the generalised system architecture, which can be reused with any cluster 
orchestrator and hosting infrastructure. A service template consumes platform-agnostic and 
platform-specific libraries. The service template is used by cluster generation scripts to create a 
package in the format required by the cluster orchestrator. Infrastructure provisioning and cluster 
provisioning are performed using separate scripts, as is the deployment of the generated package 
to the provisioned cluster. This architecture allows the flexibility to adapt each part of the system 
to different cluster orchestrator, environment and infrastructure requirements. How each module in 
Figure | is composed in the chosen Azure Service Fabric PaaS hosting model is explored in the 
following sections. 


System Components 





Figure 1 - High-level system architecture 


3.3.1 Cluster Emulator Libraries 


The libraries are split into platform-agnostic emulation libraries and platform-specific libraries. The 
platform-agnostic emulation libraries are open to reuse with any service delivery platform capable 
of hosting a .NET Core application. The platform-specific libraries, however, integrate with the 
chosen cluster orchestrator and logging solutions, coupling them to these components. Figure 2 
shows these libraries and their dependencies. 
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Cluster Emutsior Components 
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Figure 2 - Cluster emulation libraries 


ClusterEmulator.Models is a .NET Standard library which isolates service interaction models and 
only depends on Newtonsoft.Json. ClusterEmulator.Emulation is a .NET Core library which 
contains configurable emulation logic and consumes the Models library. The emulation library 
takes dependencies on .NET Core, Polly and Newtonsoft.Json. These dependencies open the library 
to reuse with any service delivery platform, infrastructure and logging solution. 


Integrations between the cluster orchestration platform and logging solutions are primarily isolated 
to the ClusterEmulator.ServiceFabric and ClusterEmulator.ApplicationInsights libraries. These 
libraries provide convenience capabilities to the service template. Template Service provides a 
service template which contains minimal code but depends upon all other project libraries and 
integrates tightly with the cluster orchestrator. 


3.3.2 ClusterEmulator.Emulation 
The emulation library contains logic for configurable service prototyping. The three parts of the 
library are; 


e loading and creation of configurable components, 
e execution of emulated service behaviours and 
e configurable components. 


The ServiceCollectionExtensions class provides methods for convenient integration with ASP.NET 


dependency injection for all parts of the library. The following sections explore the three parts of 
the library, listed above, in detail. 


3.3.2.1 Loading and Creation 


The Registry class is the central repository for step, processor, client configuration and request 
policy components. Figure 3 shows the class integrations relevant to loading and creation of 
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configured service objects. On creation, the registry is provided settings which are used to initialise 
service components through a set of injected factories. All factories share a generic IConfigFactory 
interface, accepting a setting value and returning an initialised object, allowing the abstraction of 
concrete factory types from the registry. 
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Figure 3 - Class diagram showing classes involved in registration of configured instances 


Newtonsoft.Json is used by the factories to deserialise settings. ConfigFactory handles 
unambiguous deserialisation where the setting is an explicit JSON representation of the type. 
NestedConfigFactory handles deserialisation of settings that expresses both the type to deserialise 
and its properties or a proxy of the type to create. It operates on types implementing IConfigModel 
which provides an AsTypeModel method that allows implementors to return an instance of their 
type or to return another type representing the final service component. SimpleHttpClientFactory 
is an implementation of [HttpClientFactory that does not reuse connections, enabling the creation 
of requests exhibiting socket management issues in created components. 


3.3.2.2 Execution of Emulated Behaviours 


Figure 4 describes the classes involved in the execution of emulated behaviours. The Engine class 
loads processors and the related steps from the emulation registry and encapsulates all processing 
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actions. It supports execution of a single named processor and execution of all start-up processors. 
These represent request processing and start-up or persistent service load, respectively. 


The engine executes a processor’s steps as an ordered sequence and returns failure if a step fails to 
complete. A single step can be executed multiple times in parallel. When executing in parallel, a 
task group is created and awaited. If failures occur in some tasks, the overall step failure is 
determined by the GroupClause failure clause in the step. 


The AdaptableController class provides an ASP.NET Core MVC controller. The controller exposes 
an API supporting a set of standard HTTP request verbs and calls upon the engine to execute any 
requested processor. 


CrusterEmutater Emulation Processing 
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Figure 4 - Class diagram showing classes involved in emulated processing 


3.3.2.3 Configurable Components 


The library contains four high-level configurable components. A Step encapsulates configurable 
execution logic that emulates or simulates some aspect of service business logic. A Processor 
connects steps into an ordered chain of actions. A ClientConfig describes initialisation parameters 
for HTTP client creation while a PolicyConfig is used to create resiliency policies that are 
optionally applied to HTTP requests. 
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All of the configurable service components specify deserialisation constraints on their properties 
using Newtonsoft.Json attributes. The following sections explore each component in detail, 
omitting the NewtonSoft.Json dependency from class diagrams to maintain readability. 


3.3.2.3.1 Processors 


Processors connect emulation steps to represent complex business processes within a service. 
Figure 5 shows the class dependencies relationships involved in processors. [Processor extends 
IConfigModel from the Core library, allowing the use of NestedConfigFactory to create all sub- 
types. Two sub-types enable emulation of a service with an accurate performance profile: 


e A StartupProcessor configures service-up behaviour and can be run synchronously or 
asynchronously. A start-up processor can be used to emulate initialisation actions when 
service recovery occurs or to emulate constant service load requirements through 
asynchronous execution. 


e A RequestProcessor configures per-request processing. Response payload sizes and 
simulated ingress latency are configurable. These parameters enable accurate emulation of 
network bandwidth usage and the simulation of network latency. The latency configuration 
will act as in a real system from the consumer’s side, but an issue with this approach is that 
it could increase thread pool utilisation on the processing system. An examination of 
approaches to latency configuration determined that this approach would yield the most 
accurate system representation as other approaches relied on external services. 


ClusterEmulator Emulation. Processors Class Diagram 
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Figure 5 - Class diagram of emulated processors 


3.3.2.3.2 Steps 
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Steps enable the emulation or simulation of business logic and expose a single execution method 
to consumers through the [Step interface. Figure 6 details Step class relationships. [Step extends 
IConfigModel from the Core library, allowing the use of NestedConfigFactory to create all sub- 
types, while only exposing [Step externally. 
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Figure 6 - Class diagram of emulated steps 


Four types of steps represent a broad range of service behaviours: 
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e A DelayStep allows the simulation of a known delay, typically representing a regular 
dependency that is omitted from the modelled system. 


e An ErrorStep creates a probabilistic error rate, emulating an intermittent bug in the system 
or simulating an error in a dependency omitted from the model. 


e A LoadStep generates memory usage demand and CPU load for a given period. The CPU 
usage can be configured to execute on one to all available cores. The load step enables 
emulation of per-request or constant resource demands by registering it with the 
appropriate processor. 


e A RequestStep references an HTTP client and makes HTTP requests using the client. It 
accepts service responses and manages the lifetime of requests. The RequestStep exposes 
an IRequestStep interface to allow registration of an HttpClientFactory after deserialisation 
of the core configuration. The HttpClientFactory is used to retrieve or create the required 
HTTP client. The step can be configured to fire-and-forget requests or to await them. Cache 
identification and cache uniqueness can be configured to emulate cache collision rates and 
cache contamination issues accurately. The request method and payload size can be set to 
emulate networking characteristics. 


3.3.2.3.3 HttpClientConfiguration 


The HttpClientConfiguration module provides two of the high-level configurable components, 
which are both related to ASP.NET Core HTTP client creation or usage. These are ClientConfig 
and IPolicyConfiguration and are shown in Figure 7. 


A ClientConfig describes the requirements to initialise an HttpClient and optionally references the 
names of resiliency policies which apply to all requests through the client. This abstraction allows 
the use of the default HttpClientFactory or a custom implementation to represent specific socket 
usage characteristics. As this is a simple named type, the ConfigFactory discussed in 3.3.2.1 is used 
to deserialise it. 


Polly Policies are created by types implementing the IPolicyConfiguration interface. This interface 
extends IConfigModel and restricts the return type of AsTypeModel to a Polly [AsyncPolicy 
instance. An extension class provides standardised integration with Polly for response error 
handling when creating policies. The only external interaction with policy configuration is during 
initialisation of a NestedConfigFactory. Consumers of the factory then interact with Polly 
TAsyncPolicy instances and can make use of Polly extensions and integrations with .NET Core. 


All Polly policies, at the time of writing, are represented by policy configurations: 

e FallbackConfig creates a simple fallback policy which returns the configured HTTP 
response code, reason and message content when the wrapped HTTP request returns an 
error. 

e CircuitBreakerConfig creates a simple circuit breaking policy which prevents outbound 
requests when broken when multiple successive failures occur. The break duration and 


number of faults required to cause a broken circuit are configurable. 


e AdvancedCircuitBreakerConfig creates a circuit breaking policy which triggers when the 
percentage of requests failing over a set period breaches the defined threshold. The break 


16 of 44 08 March 2020 


Performance Problems in Microservices Keith Cully 


17 of 44 


duration, failure rate threshold, sampling timeframe and minimum throughput to trigger a 
break are configurable. 


BulkheadConfig creates a bulkhead policy that restricts maximum concurrent request 
throughput with optional queueing of requests at the bulkhead. The maximum number of 
concurrent requests and allowed queue length are configurable. 


TimeoutConfig creates a timeout policy around outbound requests which allows for 
immediate cancellation of the task on reaching the allowed time or co-operative 
cancellation of delegates through a provided cancellation token. Both the timeout wait 
length, and whether or not to use co-operative cancellation are configurable. 


RetryConfig creates a policy which triggers retries on failed requests. The policy can be 
configured to retry once to infinite times. Retries can run immediately, run after a specific 
delay, follow a delay sequence or apply an exponential backoff between retry attempts. A 
randomised delay jitter can also be configured to prevent all retries from occurring with 
the same intervals. 


CacheConfig creates a cache policy that stores HTTP responses in an in-memory cache. 
The cache invalidation time is configurable and can be a relative time or an absolute time. 
The invalidation time can optionally be set as a sliding interval which causes the time to 
be reset on every access. A step making use of a cache-enabled client must set the operation 
key on the Polly Context as its cache key. 
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Figure 7 - Class diagram of HTTP communication and resilience policy configuration 


3.3.3 ClusterEmulator.ServiceFabric 


The ClusterEmulator.ServiceFabric library provides Service Fabric specific types for emulation 
engine settings, Serilog logger enrichment and extension methods for convenient registration with 
the ASP.NET Core instance container. Figure 8 shows the class diagram for this library. 
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Figure 8 - Class diagram of Service Fabric integration library 


3.3.4 ClusterEmulator.ApplicationInsights 


The ClusterEmulator.ApplicationInsights library provides Application Insights integration for 
Serilog through configuration, allowing the service code to remain decoupled from Application 
Insights. The Serilog Application Insights sink is supplemented with operation id enrichment to 
enable distributed tracing and facilitate the decoupling mentioned above. Figure 9 shows the class 
diagram of the library. The OperationIdEnricher class extracts the request ID from the Serilog event 
and adds it to a property. AppInsightsTelemetryConverter extends the Serilog converter and sets 
the operation id for Application Insights, based on the extracted property. LoggerExtensions 
extends Serilog configuration capabilities to allow JSON configuration of the new types. 
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Figure 9 - Class diagram of Application Insights library 
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3.3.5 TemplateService 


A service is the primary unit of configuration in an emulated system. This level of composition 
allows the creation of prototype systems which adhere to core microservice architecture principles 
[6], [31], [32]. The TemplateService is a concrete service implementation which integrates tightly 
with the chosen logging and orchestration platforms. It depends upon the Emulation library, NET 
Core, Application Insights, Serilog and Service Fabric. While the service integrates with Service 
Fabric, it is portable across hosting platforms, as the dependency on Application Insights is purely 
through configuration. 


The service can operate as a stand-alone service for debugging and development purposes but also 
includes .NET Core template configuration files for use in the generation of a configured Service 
Fabric Application. The use of .NET Core allows downstream build tools to avoid operating system 
dependencies. 


The Program class is the entry point for the service host process. It is responsible for loading log 
configuration, initialisation of the Serilog logger, creation of a TemplateService instance and 
registration of the service instance with the Service Fabric runtime. Figure 10 shows how the 
Program class integrates with these components. 
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Figure 10 - Class diagram of template service program integrations 
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The TemplateService class integrates the ASP.NET Core application into a Service Fabric stateless 
reliable service. A Kestrel web server [33] is initialised as a ServiceInstanceListener to enable 
service ingress and environmental context is added to the logger. Serilog is registered as a log 
provider, enabling controllers and engine code to interact with the .NET Core logging API and 
remain agnostic of Serilog. The emulation engine and its settings are registered with the 
dependency injection container. Figure 11 shows the class integrations involved. 


The TemplateController class is an ASP.NET Core MVC controller which extends the core 
AdaptableController, described in 3.3.2.2. Route and ApiController attributes are added to the class 
to inform the MVC framework to treat it as an API controller with support for specific formats. 
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Figure 11 - Class diagram of template service integrations 
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The Startup class provides ease-of-use instance container and HTTP pipeline configuration for 
ASP.NET Core applications. It initialises ASP.NET Core MVC modules, health checks and HTTP 
clients. MVC modules enable the controller serving API requests. Health checks serve lightweight 
service availability endpoints which can be consumed by an external ingress controller for request 
routing purposes. The last action of the Startup class is to call the emulation engine to execute all 
associated start-up processors. Figure 12 shows the Startup class relationships. 
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Figure 12 - Class diagram of template service start-up integrations 


3.3.6 Generation of a Configured System 


Service Fabric reliable services are deployed to clusters within application packages [34]. These 
packages consist of folders that contain an application manifest, detailing service requirements, and 
service package folders for the encapsulated services. A service folder contains a service manifest, 
configuration files, the service binaries and their dependencies. A Service Fabric cluster can be 
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composed of multiple applications, but for this project which does not have complex deployment 
orchestration requirements, the cluster contains a single application. 


The package generation pipeline in this project takes a JSON file describing a set of services as 
input and creates a Service Fabric application package. Though the pipeline is bound to Service 
Fabric, it takes no dependency on the hosting platform or the operating system executing the 
pipeline. Application generation and packaging scripts are written in PowerShell Core and use the 
cross-platform .NET Core CLI to perform build actions. The pipeline consists of application 
generation and package generation. Figure 13 shows the pipeline components, with highlighted 
tools explored in the following sections. 
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Figure 13 - Application packaging components and flow 


3.3.6.1 Application Generation 


A JSON file describes the configured application. This file details the set of services and their 
configuration settings. A schema is provided for use in application configuration files to verify 
structural correctness of the configuration, before pipeline execution. 


The AppGenerator script takes the configuration file as input. The script uses the NET CLI with 
the template service project to generate service projects matching the configuration. The script 
outputs an intermediary package containing the generated service projects, port usage information, 
an application manifest and service dependencies. In the application manifest, the default instance 
count for each service is the number of cluster nodes. This value can be overridden in the 
intermediary package if required. The intermediary package enables debugging of the generated 
services and later pipeline steps consume it. 


3.3.6.2 Package Generation 


The AppPacker script enumerates all services in the intermediary package, building service code 
packages using the .NET Core CLI and copying configuration files to their appropriate destination. 
The output is a Service Fabric application package configured as specified in the original JSON 
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configuration file. The resultant package can be deployed to a Service Fabric cluster running on 
any host. 


3.3.7 Cluster Provisioning 


The Azure Service Fabric Provisioner script, shown in Figure 14, creates resources in Azure to run 
a Service Fabric cluster. The script uses Azure Resource Manager (ARM) templates to parameterise 
resource provisioning. It accepts an ARM template, the node count, datacentre region and cluster 
tier as parameters. While this allows flexibility in the exact resources provisioned, the project also 
includes a resource template for a production-grade secure cluster. The analysis detailed in Chapter 
Error! Reference source not found. was configured using this template. 


In this configuration, a Virtual Machine (VM) Scale Set provides hosting for the Azure Service 
Fabric cluster, and a virtual network isolates the VMs from the internet. A Key Vault provides 
secure management of cluster certificates and cluster management is locked down using these 
certificates. Azure Load Balancer controls cluster ingress while Application Insights, Azure Log 
Analytics and storage are provisioned to store and process telemetry. The generated ports file 
provides parameters for configuration of the load balancer. 


The Deploy Application script, shown in Figure 14, uses the cluster management certificate to 
communicate with the cluster management endpoint. It cleans existing applications from the 
cluster, deploys the application package to the package store then registers the application. The 
cluster orchestrator then takes over and initialises the application and services. 
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Figure 14 - Cluster provisioning and deployment components and flow 
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4 Validation of Cluster Emulator 


4.1 Overview 


Cluster Emulator’s functional correctness was tested using standard continuous integration 
practices. To prove that the suite can generate a prototype which accurately represents the 
behaviour and performance profile of a real system, both a complete system and a configured 
system modelling it were created. 


Three test phases were executed against each system to evaluate their soundness and draw 
comparisons between their performance profiles. The first phase tested the configuration accuracy 
on a local deployment. The second phase evaluated the soundness of each system, operating in a 
distributed cloud environment. The third phase evaluated how accurately the load profile of the 
prototype system emulated that of the real system. The following sections describe the systems and 
the test phases that were executed. 


4.2 Functional Correctness 


Unit tests, using the MSTest [35] framework, validate the source code integrity for all projects. A 
continuous integration pipeline on Azure DevOps [36] builds the source code and executes all unit 
tests for every change pushed to the GitHub repository. While development was performed on 
Windows, the automated build and tests execute on Linux, yielding expanded coverage of 
development environments. Acceptance checks were manually executed against the template 
service sample configuration, deployed to a local Service Fabric cluster, using Postman [37]. 


4.3 Comparison System 


The comparison system consists of a single Service Fabric application running three stateless 
services hosting ASP.NET Core applications with different performance profiles and relying on 
service-to-service communication to fulfil user requests. Figure 15 shows the cluster topology and 
service interactions involved. 
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Figure 15 - Proof-of-concept service interactions 
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e Random Generator exposes an interface to retrieve a random number. Internally, it 
generates prime numbers to use as seed values for random number generation. 


e Name Lookup serves GET and POST requests to retrieve a name or list of names, based 
on numeric identifiers. It maintains a dictionary of keys and names in-memory and 
performs dictionary lookups for requested keys. Not all keys are populated with 
corresponding name values. Requests for a single lookup fail if there is no name populated 
but requests for multiple lookups return whatever matching values are present. 


e Name Generator returns random names to a consumer. It allows querying for one to many 
names. When querying multiple names, all requests to the Random Generator service are 
executed in parallel. If any of the parallel requests fail, the call returns, otherwise a single 
request is made to the Name Lookup service. The request will be GET if a single name is 
required or POST for multiple names. 


Though the system is small, it contains CPU bound operations, memory-intensive services, service 
integrations, parallel request management and uses multiple HTTP methods. This variance makes 
it sufficiently complex to verify the soundness of a prototype system delivered through Cluster 
Emulator while remaining explainable. 


4.4 Configured System 


The configured system maintains the same topology as the real system but emulates the services’ 
business logic. The average length of names is eight characters and the average batch size for multi- 
name requests is ten. These values determine payload sizes, memory usage and concurrency 
requirements. Artificial latency is set to zero since the topologies and infrastructure are identical. 


e Random Generator is configured with a single request processor representing retrieval of 
arandom value. This depends on a load step which emulates the single-threaded CPU load 
of prime and random generation in the real service. 


e Name Lookup is configured with two request processors. One contains an error step with 
low probability, emulating key miss errors, and a load step to emulate cache access and 
memory usage. The other processor wraps a single load step which executes ten times in 
parallel for each processor execution. This concurrency emulates the real service, which 
splits cache retrieval into parallel operations for multi-lookup requests. 


e Name Generator is configured with two clients, one for making requests to each of the 
other services. It contains two request processors to emulate service behaviour for retrieval 
of single or multiple names. Each of these processors uses request steps which call the 
appropriate Random Generator API then Name Lookup API in order. The request step to 
emulate retrieval of multiple random IDs executes ten times in parallel with any failure 
causing the step to report a failure and the processor to return. 


4.5 Configuration Accuracy 


Initial setting values for the configured system derive from the system design. As such, some 
assumptions were made about data sizes and resource requirements. End-to-end tests were executed 
against each system running on a local cluster to establish the level of inaccuracy in the assumptions 
made and enable correction of errors. The cluster was reset before each system test, to prevent 
interference between deployed systems. 
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4.5.1 Test Environment 


Cluster Location Local Cluster 


Cluster Orchestrator Service Fabric 


Cluster Size One node running on a physical machine 

Processor Intel Core i17-6600U CPU @ 2.60GHz, 2 cores, 4 logical processors 
Physical Memory 8GB 

Operating System Windows 10 Pro 


Software Framework NET Core 2.2 


Test Harness Postman 
Table 2 - Configuration accuracy test environment 


4.5.2 Tests 


Two call profiles were created and reused throughout all test phases. These call profiles were 
executed as requests through Postman, as described in Appendix A. 


e Get Name — Execute a GET request against the Name Generator API to retrieve a single 
name. 


e Get Ten Names — Execute a GET request against the Name Generator API to retrieve ten 
names. 


4.5.3 Results 


Average name length was overestimated, with response sizes being up to one hundred per cent 
larger in the configured system. Processor requirements for the Name Lookup service were 
overestimated. This inaccuracy caused user response times from the configured system to be fifteen 
per cent longer than those of the comparison system. 


To correct this, both response sizes and processor requirements were adjusted. These adjustments 
aligned both data sizes and response times to within ten per cent of the real system. As the local 
environment was prone to resource contention issues, this was sufficiently accurate to move on to 
testing in a distributed environment. 


4.6 Distributed Operation 


This phase set out to verify that test configuration, environment and request routing were working 
as expected in a distributed environment. The number of instances of each service was increased 
to three and the applications were each deployed to three-node clusters in Azure. Apache JMeter 
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[38] was used to execute a set of load tests with user results, logs, and performance metrics analysed 
for issues. 


4.6.1 Host Environment 


As the goal was to evaluate the soundness of the systems, and since infrastructure templates did not 
exist at this stage, the minimum infrastructure to secure and monitor clusters was provisioned, as 
described in Table 3 and Figure 16. Along with log data emitted from services, Application Insights 
captures primary processor and network usage metrics. These data points were sufficient to 
evaluate the platform. The initial infrastructure consisted of a Service Fabric cluster using Azure 
Load Balancer to route inbound traffic, Key Vault to store secrets and Application Insights to gather 
telemetry. After identifying external traffic sources during testing, a virtual network was added to 
secure the nodes. 


Cloud Provider Azure 

Data Centre Region North Europe 

Cluster Orchestrator Service Fabric 

Cluster Size Three nodes running on virtual machines 
Virtual Machine Type §Standard_D1_v2 


Operating System Windows Server 2016 (WindowsServerSemiAnnual, Datacenter- 
Core-1803-with-Containers-smalldisk) 


Software Framework .NET Core 2.2 


Ingress Controller Azure Load Balancer 


Table 3 - Distributed operation host environment 


28 of 44 08 March 2020 


Performance Problems in Microservices 





«AT IBLORIB MAN Re 


Load Balancer 


ingress comrodeny 





weness tests 


Handles serve ingress wth 


Test Infrastructure 


eAzumSenicehabre» 


Service Fabric Cluster 


13 Nese chai) 


| «AzuereXeA’auts 


| Key Vault 


@) 


{secret storage? 





Figure 16 - Distributed operation infrastructure diagram 


4.6.2 Client Environment 


Number of Machines One 


Processor 


Physical Memory 8GB 


Operating System 


Test Harness 


Table 4 - Distributed operation client environment 


4.6.3 Load Tests 


JMeter 5.1.1 


Windows 10 Pro 


Keith Cully 


+ AZUTOV TURP NINeS CHETAN 


VM Scale Set 


re) 


envice hosting 


«AZ UMADINE aton/ ns» 


Application Insights 


- 
(App and annie node telemetry) 


Intel Core i17-6600U CPU @ 2.60GHz, 2 cores, 4 logical processors 


Four load tests were defined in JMeter, reusing the call profiles from the previous test phase. The 
integrity of the systems and deployed environment were evaluated using Load test 1. The entire 
test suite was then used to determine if any load-related throttling issues could occur. Table 5 


describes these load tests. 


Load Test Load 
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1 Steady load of two requests per second for the Get Five Five 
Single Name profile and one request per second for the seconds. minutes. 
Get Ten Names profile. 


2 Steady load of six requests per second for the Get Single Fifteen Five 
Name profile and three requests per second for the Get seconds. minutes. 
Ten Names profile. 


3 Steady load of eighteen requests per second for the Get Forty Five 
Single Name profile and nine requests per second for seconds. minutes. 
the Get Ten Names profile. 


4 Steady load of fifty-four requests per second for the Get Ninety Five 
Single Name profile and twenty-seven requests per seconds. minutes. 
second for the Get Ten Names profile. 


Table 5 - Distributed operation load tests 


4.6.4 JMeter Configuration 


A JMeter test plan was created containing user-defined variables to allow reuse across each load 
test, with appropriate variable updates applied. The test plan contained a DNS Cache Manager 
config element, two thread groups for the configured system, two thread groups for the real system 
and a result view listener. The configured system thread groups started after a delay to prevent 
overlap with real system tests. Appendix A.2 details the JMeter environment configuration. 


The DNS Cache Manager used the Cloudfare DNS resolver [39] allowing any load balancer 
configuration to be honoured. Each thread group contained an HTTP Request Default element 
setting the domain and port, an HTTP Cache Manager Element to clear caches between requests, a 
Throughput Timer to control request rate and an HTTP Request element. The HTTP Request 
element completes the Get Name or Get Names API request path and contains an HTTP Header 
Manager to generate a unique request Id. 


4.6.5 Results 


The tests identified several issues which would have invalidated the systems’ performance profiles. 
These ranged from configuration errors through inconsistent request routing and external attempts 
to compromise the clusters. The following section describes these issues and their corresponding 
fixes. 


Application Configuration 


The simulated failure rate for Get Name request processing in the configured Name Lookup service 
was double that of the real service. The response time for these failures was also significantly lower 
in the configured system. Both issues were due to misconfiguration. The configured error 
probability was double the real failure rate and the order of steps in the configured request processor 
caused the failure to occur earlier than it should have. The error probability was adjusted to the 
failure rate of the real service and the order of steps in the processor was corrected. 


Code Error 
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An investigation into logs revealed that exceptions were occasionally occurring during processing 
of Get Names requests in the real system. The cause was a simple code error which was remediated. 


Throttling of Logs 


During the investigation of failures in the real system, some log traces could not be found. The 
volume of log data was significantly lower than expected. Adaptive sampling was enabled for the 
real services causing Application Insights to throttle logging. As the reduced network traffic could 
have skewed the performance profile, adaptive sampling was disabled. 


Malicious External Traffic 


At the outset of this test, the virtual machines were exposed on a public IP address. Soon after 
deploying the clusters, inbound requests attempting to access server resources began to occur. The 
requests were targeting administrative resources for various server technologies and originated 
from several countries. The cluster needed to be secured to prevent any compromise of the systems 
and remove this uncontrolled load. A virtual network was created and new cluster infrastructure 
was provisioned inside the virtual network. The load balancer was then configured to only forward 
traffic matching specific ports and protocols to the appropriate service ports within the network. 


Routing of Requests 


Conversely to local test results, the configured system yielded much faster response times than the 
real system. Both the real and configured systems contained issues which led to this behaviour. In 
the configured services the base URIs for service-to-service request steps were using localhost as 
their base domains, leading to improved performance but reduced resiliency. In the real services, 
the service URIs were being resolved once, on service start-up, using the Service Fabric 
ServicePartitionResolver. This static resolution led to only one VM instance receiving traffic for 
internal service requests, reducing the capacity of the system and impacting performance. 


Several platform-specific options to resolve these routing issues were explored. These included 
Service Fabric DNS Service [40], Service Fabric Reverse Proxy [41], and Service Fabric Name 
Service [42]. However, to both minimise potential bugs due to custom implementation and simplify 
end-user configuration, direct DNS lookups were the preferred option. This fix required updating 
both systems to specify the full URIs in their setting files and configuration of the load balancers 
to forward traffic for the Random Generator and Name Lookup services. 


Load Balancer Load 


At the outset, the load balancer was only configured for the Name Generator service. Liveness tests 
were executing requests requiring heavy end-to-end processing from the service and its 
dependencies. This configuration was creating an artificial load on the systems and is contrary to 
best practice for liveness tests. All services were updated to include health check middleware, 
adding a simple liveness status page and the load balancer configuration was updated to target these 
health endpoints for service liveness tests to address this. 


4.7 Load Profile Comparison 


Once all known issues with each system and infrastructure configuration were addressed, a 
systematic comparison of each system’s load profile was performed. This set out to evaluate how 
accurately the generated system represented the real system. 
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4.7.1 Test Configuration and Measurement 


This evaluation reused the tests, test configuration, client, and environment configuration from the 
previous phase. Clusters were deleted and provisioned anew, to ensure the test environments were 
clean. A minimum ten-minute cooldown period was enforced between load tests to allow machines 
to return to their idle state. 


Azure Resource Monitor was used to track system resource usage, with graphs generated in real- 
time. User responses were exported from JMeter and service results were extracted from 
Application Insight logs. User and service data were then processed in Microsoft Excel to generate 
the tables and graphs presented herein. Testing of each system was staggered by five minutes to 
avoid overlap, but this time difference is offset in generated charts to allow visual comparison of 
trends. 


4.7.2 Results 


The performance profiles align well and follow the same general trends. All charts represent the 
configured system as Emulated (E) and the real system as Comparison (C). Appendix A.3 presents 
tabular result data. 


Client Responses 


Figure 17 shows the client response times for each system and user profile. These data are presented 
in bar form as fifteen-second aggregates and linear form as a moving average over one minute. 
Trend lines, following an exponential fit for each dataset, are mapped to the chart. Through each 
load test, average response times closely matched, while performance degradation in both systems 
followed the same trend. 
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Figure 17 - Proof-of-concept client response time comparison 
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Figure 18 shows the count of each response code, returned to the client, aggregated over fifteen- 
second intervals and the one-minute moving average. Both systems exhibited similar response and 
error rates throughout each load test. 
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Figure 18 - Proof-of-concept client response counts 


Figure 19 shows the client thread pool size for each test plan. A client-side issue caused some thread 
contention at the end of the final load test targeting the real system. This issue led to a dip in request 
volume targeting the real system for the last minute of the test. As the trends align so closely and 
would match more accurately if the final minute were omitted, the issue has no statistical impact 
on the result of this comparison. 
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Figure 19 - Proof-of-concept client thread counts 


Service Responses 


Figure 20 through Figure 22 show the response times for each service response in the systems. To 
prevent the client-side issue detailed above from skewing the trendline for the Random Generation 
service, the last minute of incomparable data is omitted from the charts. All trendlines follow an 
exponential fit of their data points. 
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The performance trends of each service request match closely between both the real and configured 
system. The most extensive variance was observed in the Random Generation service though the 
trend remained close enough to demonstrate that the configured system was a valid prototype-level 
representation. 
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Figure 20 - Name Generator response times 
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Figure 21 - Random Generator response times Figure 22 - Name Lookup response times 


System Performance 


Figure 23 through Figure 26 show the CPU and bandwidth usage of the systems through each of 
the load tests. Apart from one outbound networking anomaly during load test two, the processor 
and bandwidth requirements of both systems were very closely aligned. This demonstrates that 
while accurately representing user and service response times, the generated system also accurately 
recreates resource utilisation requirements of the modelled system. 
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Figure 23 - Load test I system metrics comparison (Left — Configured, Right — Real) 
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Figure 24 - Load test 2 system metrics comparison (Left — Configured, Right — Real) 
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Figure 26 - Load test 4 system metrics comparison (Left — Configured, Right — Real) 
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5 Conclusion and Future Work 


5.1 Conclusion 


A flexible and extensible system for rapid prototyping of microservice applications, deployed to 
and orchestrated within secure cloud-hosted clusters was designed, developed and evaluated. The 
prototyping suite was shown to be capable of creating an accurate representation of a real system, 
with performance profiles and operational characteristics of a real application and a configured 
application aligning closely. 


5.2 Further Development 


Cluster Emulator is a modular prototyping suite which can facilitate integration with the various 
existing microservice platforms and allow adaption to new technologies. It has been integrated with 
Service Fabric and Azure to facilitate testing in this project but is open to extension. The core 
emulation libraries focus on a narrow field of microservice communication. The following section 
details extensions which would expand the capabilities of the library and the combined suite. 


5.2.1 Publish-Subscribe Pattern 


The publish-subscribe pattern is a messaging pattern which allows the notification of events to 
registered consumers in an asynchronous manner. While Cluster Emulator is flexible enough to 
allow approximate modelling of this pattern, it would be a rigid implementation which could not 
deal with new subscribers. As this is a typical pattern in distributed system design, explicit support 
for this messaging pattern would increase the breadth of use cases significantly. 


5.2.2 Fallback Policy Improvement 


The Fallback Policy type only supports returning a specific response. A common approach to 
dealing with outages in dependencies that can be updated asynchronously is to add pending updates 
to a queue for later processing. The Fallback Policy type could be updated to support triggering a 
separate action or chain of actions to support this scenario. 


5.2.3 Docker Integration 


Docker [24] is the leading platform for automating the deployment of containerised applications. 
A service template and scripts to enable integration with Docker could be added to Cluster 
Emulator. This extension would broaden the suite’s potential applications to many other cluster 
orchestrators. 


5.2.4 Kubernetes Integration 


As explored in Chapter 3.2.3, Kubernetes is the other leading cluster orchestrator in use and works 
with Docker containers. Once Docker support is added to Cluster Emulator, deployment scripts for 
generation of a Kubernetes orchestrated cluster could be added to support this platform. 


5.2.5 Service Mesh Integration 


A potential solution to minimising the complexities involved in adding resilience to microservices 
is to utilise a service mesh, as described in [43]. Istio [44] is a widely used service mesh technology. 
Support for Istio could be added to Cluster Emulator to enable the examination of performance 
properties when utilising a service mesh. 
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Appendix A Cluster Emulator Evaluation 
A.1 Building and Deploying the Applications 


All projects and service configuration files for the system validation phase are available on this 
GitHub branch: https://github.com/K-Cully/performance-of-microservices/tree/proof-of-concept. 





The real Service Fabric application is defined under /ComparisonApplication. The configured 
application is found under /ClusterEmulator. Both solutions can be built and deployed to Azure 
using standard Service Fabric tooling. 


Packaging and local installation of the configured application can also be performed using the 
package, install and remove PowerShell scripts included. Service configuration files are located in 
ServiceName/PackageRoot/Config/Settings.xml files. 


A.2 Testing the Applications 


PostMan request files, used as local acceptance tests, are available in the supplementary submission 
data. These can be imported and executed in the latest version of PostMan but may require 
hostname updates depending on the deployed cluster name. 


JMeter Configuration files for the system validation tests are available here: https://github.com/K- 
Cully/performance-of-microservices/tree/proof-of-concept/test_plans. The tests can be opened and 
executed in JMeter 5.1.1, with hostname adjustments to suit the deployed cluster. Load tests were 
executed through the JMeter command-line interface, with Java Virtual Machine heap flags “- 
Xms1280m -Xmx1280m -XX:MaxMetaspaceSize=256m”’ set, to maximize memory availability. 











A.3 Aggregated Result Data 


Respone Codes 


200 204 418 500 Total Count Total Duration 
Service Calls Count Duration Count Duration Count Duration Count Duration 
NameGenerator 
GetName 
Comparison 2032 7.378 186 9.62s 2218 7.56s 
Emulated 2025 6.6s 208 7.515 2233 6.69s 
GetNames 
Comparison 810 10.69s 5 100.3s 815 11.24s 
Emulated 711 = 10.91s 2 102.76s 713 11.17s 
NameLookup 
LookupName 
Comparison 1992 0.81ms 176 0.67ms 2168 0.8ms 
Emulated 2025 1.42ms 207 0.96ms 2232 1.38ms 
LookupNames 
Comparison 788 2.78ms 788 2.78ms 
Emulated 711 2.04ms 711 2.04ms 
RandomGenerator 
GetRandom 
Comparison 10252 1.66s 10252 1.66s 
Emulated 9417 2.11s 9417 2.11s 


Table 6 - System Validation service results 
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Success Fail Avg Elapsed StdDev Elapsed 


Load Test 1 
Get Name (C) 217 17 2.6s 956.3ms 
Get Name (E) 196 23 2.7s 1.1s 
Get Ten Names (C) 98 O 3.1s 888.7ms 
Get Ten Names (E) 95 0 3.2s 876.6ms 
Load Test 2 
Get Name (C) 500 45 3.2s 1.1s 
Get Name (E) 418 44 3.85 2.7s 
Get Ten Names (C) 191 0 4.7s 1.5s 
Get Ten Names (E) 132 0 6.85 3.1s 
Load Test 3 
Get Name (C) 713 59 6.6s 4.2s 
Get Name (E) 776 70 6.0s 4.6s 
Get Ten Names (C) 258 O 10.0s 7.2s 
Get Ten Names (E) 226 O 11.4s 8.6s 
Load Test 4 
Get Name (C) 693 79 18.4s 17.2s 
Get Name (E) 790 91 16.1s 14.7s 
Get Ten Names (C) 290 5 24.1s 21.2s 
Get Ten Names (E) 322 2 21.9s 18.5s 


Table 7 - System Validation user results 
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Appendix B Building Cluster Emulator 


Clone the repository https://github.com/K-Cully/performance-of-microservices/. The Cluster 
Emulator projects are built with standard, publically documented, .NET CLI build tools. Version 
2.2 .Net Core SDK must be installed. Running “dotnet build” then “dotnet test” will result in 
building the projects and executing all unit tests. 





To debug a cluster with a sample service configuration applied, the Service Fabric SDK must be 
installed. In Visual Studio 2019, open the ClusterEmulator solution, set ClusterEmulator as the 
target project and start a debugging session (F5). As all projects and tooling follow standards, 
documentation is available on the internet to debug the generated Service Fabric application in 
other environments if required. 


To create an application with custom configuration, refer to the configuration guidance on GitHub. 
(https://github.com/K-Cully/performance-of- 
microservices/blob/master/ClusterEmulator/ClusterEmulator.Emulation/A pplicationAndServices. 
md). Once a valid configuration file is created, it can be deployed to Azure by following the steps 
detailed in Error! Reference source not found. or can be packaged for deployment to any Service F 
abric cluster using these steps: 








1. Install the .NET Core 2.2 SDK. 

2. Install and launch PowerShell Core. 

3. Install the Service Fabric PowerShell modules: 
Install-Module -Name Microsoft.ServiceFabric.Powershell.Http -AllowPrerelease -Scope 
CurrentUser 


4. Open a PowerShell Core session in the ClusterGeneration folder of the repository. 


5. Run “.\AppGenerator.psl -Name <ApplicationName> -ConfigFile 
<AbsolutePathToConfigFile>” 


6. Run “.\AppPacker.psl -Name <ApplicationName>” 
7. This will generate a folder at <repositoryRoot>\generated\<ApplicationName>\pkg. 


This folder is a Service Fabric application package which can be deployed to any Service 
Fabric cluster. 
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